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Controlling  interacting  systems  in  noisy  environments 

ABSTRACT 

The  objective  of  this  project  is  to  develop  concepts  for  the  analysis  of  the  dynamics  of  interacting  systems  in  a  noisy  environment.  New 
approaches  should  lead  to  a  better  understanding  of  system  dynamics  and  generate  novel  efficient  algorithms  of  stochastic  optimal  control 
for  interacting  systems. 

One  of  the  central  issues  that  we  address  is  dynamics  of  noise- induced  switching.  The  phenomenon  underlies  a  large  portion  of  all 
significant  changes  that  occur  in  systems  in  noisy  environment.  Examples  range  from  breakdown  events  in  complex  systems  to  swarming  in 
systems  of  interacting  vehicles  to  overcoming  barriers  by  such  vehicles.  Therefore  understanding  the  switching  dynamics  is  instrumental  for 
developing  highly  efficient  ways  of  controlling  noisy  systems. 

Central  to  the  theoretical  approach  is  the  notion  that  the  dynamical  trajectories  followed  in  switching  fonn  narrow  tubes.  We  demonstrate 
that  the  tubes  can  be  directly  observed  in  experiment.  Quantitatively,  the  tubes  are  characterized  by  the  distribution  of  trajectories.  To  find  it 
theoretically  we  modify  the  instanton  technique  developed  in  a  completely  different  area,  the  quantum  field  theory.  This  approach  maps  the 
problem  of  most  probable  switching  trajectories  in  noisy  dissipative  systems  onto  a  problem  of  Hamiltonian  dynamics  of  an  auxiliary  system 
of  a  higher  dimension. 
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1  Foreward 


Developing  mathematical  tools  for  describing  the  collective  motion  of  multi-agent  systems 
is  critical  for  new  approaches  that  should  enhance  Army  operations,  since  such  tools  should 
ultimately  lead  to  efficient  means  of  controlling  the  collective  motion.  Multi-agent  systems 
originate  from  various  applications,  from  biology  to  physics  to  transportation  to  multi-robot 
systems.  They  display  complex  behavior  [14,  25,  7,  37],  with  pattern  formation  and  swarming 
as  observed  in  biological  populations  including  bacterial  colonies  [1,  3,  4],  slime  molds  [22,  27], 
locusts  [13]  and  fish  [8].  Mathematical  studies  of  this  behavior  have  been  performed  for  a  few 
decades.  The  information  gained  from  these  studies  has  already  led  to  an  increased  ability 
to  intelligently  design  and  control  man-made  vehicles  [9,  19,  21,  26,  35].  The  mathematical 
models  can  capture  emergent  properties  of  numerous  existing  and  future  applications  of 
military  and  industrial  platforms. 

Several  types  of  mathematical  models  have  been  used  to  describe  coherent  patterns  and 
swarms.  One  popular  approach  is  based  on  a  continuum  approximation  in  which  scalar  and 
vector  fields  are  use  d  for  the  relevant  quantities  [13,  16,  32,  33,  34],  Another  popular  ap¬ 
proach  is  based  on  treating  every  individual  or  object  as  a  discrete  particle  [8,  15,  16,  33,  36]. 
Such  many-particle  systems  typically  have  their  own  dynamics,  but  interact  with  others.  In 
many  programs  of  DoD  interest,  interacting  particles  in  external  static  or  time-dependent 
potentials  is  a  central  theme  in  the  construction  and  analysis  of  organized  behavior.  Dy¬ 
namically  interacting  autonomous  swarmer  (DIAS)  systems  are  comprised  of  a  multitude  of 
simple  autonomous  vehicles,  which  are  loosely  coupled  via  communication.  They  will  play 
a  key  role  in  future  deployments,  as  the  drive  to  miniaturize  electronic  devices  results  in 
smaller  and  more  capable  self-mobile  machines  with  limited  decision  making  abilities. 

A  basic  physical  principle  is  that,  as  systems  become  smaller,  an  increasingly  important 
role  is  played  by  external  fluctuations  in  the  environment.  Interacting  particles  subject  to 
external  fluctuations  but  coupled  through  communication  needs  to  be  understood  from  a 
point  of  view  of  stochastically  modulated  many-body  dynamical  systems.  A  major  math¬ 
ematical  challenge  comes  from  the  fact  that  the  regular  motion,  without  fluctuations,  has 
multiple  attractors,  and  fluctuations  cause  inter-attractor  switching,  thus  bringing  additional 
time  scales  associated  with  the  switching  rates.  The  delicate  interplay  of  interaction,  fluctu¬ 
ations,  and  multi-stability  provides  a  foundation  for  pattern  formation.  Understanding  this 
interplay  is  a  key  to  the  control  of  the  many-particle  dynamics.  Analysis  of  stochastic  inter¬ 
acting  dynamical  systems  would  lead  to  streamlined  computer,  communication,  surveillance, 
and  reconnaissance  systems. 

This  research  couples  the  rapidly  developing  areas  of  nonlinear  phenomena,  fluctuations, 
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and  dissipative  systems  far  from  thermal  equilibrium.  A  key  concept  in  the  analysis  of  mo¬ 
tion  is  noise-activated  escape  from  a  metastable  state  of  local  equilibrium,  e.g.  a  state  at 
the  minimum  of  a  potential  well.  Such  escape  leads  to  an  interstate  switching.  Activated 
processes  underlie  many  fundamental  phenomena  in  nature,  such  as  diffusion  in  solids,  nu- 
cleation,  and  protein  folding.  Much  less  is  known  about  switching  of  systems  away  from 
thermal  equilibrium,  especially  those  driven  by  time-dependent  potentials  used  in  swarming 
and  pattern-forming  models.  It  is  poorly  understood  whether  escape  rates  of  nonequilibrium 
systems  should  display  any  universal  scaling  dependence  on  control  parameters  at  all.  Gain¬ 
ing  an  insight  into  escape  in  driven  systems  requires  that  one  knows  how  a  system  moves 
in  an  activated  process.  Even  though  an  activated  process  happens  at  random,  when  it 
does  happen  the  system  is  most  likely  to  follow  a  certain  dynamical  path  [18,  10,  24,  28,  5]. 
Knowing  this  path  opens  the  way  for  control  over  activated  processes.  Such  knowledge  is 
useful  for  random  clustering  of  dynamical  patterns.  Much  of  the  motivation  comes  from  our 
recent  evidence  that  activated  processes  that  induce  escape  can  be  selectively  controlled. 
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1  (a)  Switching  probability  distribution  in  a  parametrically  driven  micro-electro- 

mechanical  oscillator.  The  probability  distribution  P\2{X,  Y )  is  measured  for  switch¬ 
ing  out  of  state  A\  into  state  A 2.  (b)  The  peak  locations  of  the  distribution  are 
plotted  as  black  circles  and  the  theoretical  most  probable  switching  path  is  indi¬ 
cated  by  the  red  line.  All  trajectories  originate  from  within  the  green  circle  in  the 
vicinity  of  A\  and  later  arrive  at  the  green  circle  around  Aq.  The  portion  of  the 
distribution  outside  the  blue  lines  is  omitted .  8 

2  Snapshots  of  a  swarm  taken  at  (a)  t  =50,  (b)  t  =60,  (c)  t  =62,  (d)  t.  =64,  (e)  t.  =66, 

(f)  t  =68,  (g)  t  =70,  (h)  t  =72,  (i)  t  =74,  and  (j)  t  =76,  with  N=300  particles,  and 
noise  intensity,  D=0.08.  The  swarm  was  in  a  rotational  state  when  the  time  delay 

of  1  was  switched  on  at  t  =40 .  9 

3  Scaled  ratios  of  the  partial  work  variance  to  mean  partial  work  for  fluctuations 

about  periodic  attractors  of  a  resonantly  modulated  Duffing  oscillator,  as  functions 
of  the  reduced  squared  modulation  amplitude  (3.  The  curves  1  and  2  refer  to  large- 
and  small- amplitude  vibrations,  respectively.  The  functions  Si  2  diverge  at  the 
corresponding  bifurcation  points  of  the  oscillator .  9 

4  Phase  portrait  of  a  two- variable  system  with  two  stable  states  A\  and  A 2.  The 
saddle  point  S  lies  on  the  separatrix  that  separates  the  corresponding  basins  of 
attraction.  The  thin  solid  lines  show  the  downhill  deterministic  trajectories  from  the 
saddle  to  the  attractors.  A  portion  of  the  separatrix  near  the  saddle  point  is  shown 
as  the  dashed  line.  The  thick  solid  line  shows  the  most  probable  trajectory  that  the 
system  follows  in  a  fluctuation  from  A\  to  the  saddle.  The  most  probable  switching 
path  (MPSP)  from  A\  to  A2  is  comprised  by  this  uphill  trajectory  and  the  downhill 
trajectory  from  S  to  A2.  The  plot  refers  to  the  system  studied  experimentally,  an 


underdamped  nonlinear  parametrically  modulated  oscillator  with  the  modulation 

frequency  close  to  twice  the  eigenfrequency .  10 

5  A  snapshot  of  the  probability  p(X)  near  the  extinction  plane  X2  =  0  for  the  SIS 


model;  p  is  quasi-continuous  in  X\  /N,  with  X\  and  X2  being  the  total  numbers 
of  susceptibles  and  infected,  respectively,  and  N  being  the  charateristic  total  pop¬ 
ulation.  The  data  of  simulations  refer  to  pt  =  9,  Rq  =  4,  p!  =  p/(p  +  k)  =  0.1, 
where  p  is  the  birth-death  rate  and  Rq  is  the  infection  reproduction  rate.  For  t  =  0 
the  system  was  at  the  stabl;e  state  X^4,  the  total  number  of  particles  was  N  =  50. 

(b)  Asymptotic  optimal  trajectories  for  extinction  for  p  — »  0  (solid  line)  and  p  =  0 
(dashed  line) .  11 
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6  The  switching  exponent  Q  for  the  SIS  model  of  epidemics.  The  epidemics  extinction 
rate  is  IT  oc  exp(— Q).  The  solid  and  dashed  lines  show  the  results  for  the  birth- 
death  rate  fi  — >  0  and  /i  =  0,  respectively.  The  data  points  are  obtained  from  the 


numerical  solution  of  the  master  equation  for  the  total  initial  populations  N  =  50 
and  N  =  100,  which  made  it  possible  to  directly  extract  the  exponent  Q .  11 


7  Extinction  factor  increase  as  a  function  of  vaccination  parameters  numeri¬ 
cally  computed  using  the  Monte  Carlo  simulation.  Parameters  shown  are 
g  ^vaccination  amplitude  and  v  ^vaccination  frequency.  The  population  size 
is  N  =  1000 .  12 
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2  Statement  of  the  problem  studied 


The  objective  of  this  project  is  to  develop  concepts  for  the  analysis  of  the  dynamics  of 
interacting  systems  in  a  noisy  environment.  New  approaches  should  lead  to  a  better  under¬ 
standing  of  system  dynamics  and  generate  novel  efficient  algorithms  of  stochastic  optimal 
control  for  interacting  systems. 

One  of  the  central  issues  that  we  address  is  dynamics  of  noise-induced  switching.  The 
phenomenon  underlies  a  large  portion  of  all  significant  changes  that  occur  in  systems  in 
noisy  environment.  Examples  range  from  breakdown  events  in  complex  systems  to  swarm¬ 
ing  in  systems  of  interacting  vehicles  to  overcoming  barriers  by  such  vehicles.  Therefore 
understanding  the  switching  dynamics  is  instrumental  for  developing  highly  efficient  ways  of 
controlling  noisy  systems. 

Central  to  the  theoretical  approach  is  the  notion  that  the  dynamical  trajectories  followed 
in  switching  form  narrow  tubes.  We  demonstrate  that  the  tubes  can  be  directly  observed  in 
experiment.  Quantitatively,  the  tubes  are  characterized  by  the  distribution  of  trajectories. 
To  find  it  theoretically  we  modify  the  instanton  technique  developed  in  a  completely  different 
area,  the  quantum  field  theory.  This  approach  maps  the  problem  of  most  probable  switching 
trajectories  in  noisy  dissipative  systems  onto  a  problem  of  Hamiltonian  dynamics  of  an 
auxiliary  system  of  a  higher  dimension. 

This  award  facilitated  the  formation  of  the  new  team  of  investigators  consisting  of  an  ap¬ 
plied  mathematician,  a  theoretical  physicist,  and  an  experimental  physicist.  It  also  included 
collaboration  with  applied  mathematician  Ira  Schwartz  at  the  Naval  Research  Laboratory. 
Utilizing  the  complementary  skill  sets,  the  group  produced  significant  results  in  physics, 
mathematics,  and  the  life  sciences. 

3  Summary  of  the  most  important  results 

The  following  is  a  brief  summary  of  the  most  important  results: 

1.  We  have  solved  the  long-standing  problem  of  noise-induced  switching  in  periodically 
modulated  systems.  We  found  the  distribution  of  trajectories  followed  in  switching. 
This  distribution  may  display  several  peaks  separated  by  the  modulation  period.  An¬ 
alytical  results  agree  with  the  results  of  simulations  of  a  Brownian  particle  in  a  model 
modulated  potential. 

2.  We  have  solved  the  problem  of  control  of  distribution  over  period- two  states.  We  show 
that  a  comparatively  weak  field  can  strongly  affect  the  rates  of  switching  between  the 
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states.  The  logarithm  of  the  rate  change  is  linear  in  the  control  field  amplitude.  We 
predict  lifting  of  degeneracy  of  period-two  states  and  the  possibility  of  an  extremely 
strong  fluctuation-mediated  wave  mixing. 

3.  To  interpret  switching  events  in  the  driven  colloidal  system,  it  is  necessary  to  track  each 
object  in  an  ensemble  of  nominally  identical  objects.  We  have  developed  algorithms 
to  identify  large  fluctuations  in  the  space  of  particle  coordinates.  The  method  forms 
a  predictor  conditioned  on  the  existing  flow  and  Gaussian  noise  amplitude,  then  flags 
events  with  low  probability.  The  algorithm  searches  for  correlations  among  the  particles 
that  precede  switching  events.  The  approach  can  be  extended  to  include  non-Gaussian 
noise  and  periodic  modulation  of  the  optical  field. 

4.  We  have  made  significant  progress  in  developing  a  general  mathematical  approach  to 
the  analysis  of  switching  in  systems  driven  by  a  Gaussian  and  a  non-Gaussian  noise. 
Our  preliminary  results  indicate  the  possibility  of  exponentially  strong  effect  of  a  non- 
Gaussian  noise  on  the  switching  rate.  The  effect  can  be  expressed  in  a  closed  form 
in  terms  of  the  characteristic  functional  of  the  noise,  which  is  important  for  many 
applications.  We  have  started  studying  specific  important  types  of  non-Gaussian  noise, 
and  in  particular,  shot  noise.  Analytical  results  agree  with  the  results  of  simulations 
of  a  Brownian  particle  in  a  model  modulated  potential. 

5.  We  have  developed  a  formulation  that  allows  one  to  observe  switching  trajectories 
and  find  the  most  probable  paths  without  making  any  preliminary  assumptions  about 
the  system.  This  formulation  has  been  tested  experimentally  using  a  mesoscopic  sys¬ 
tem  of  significant  interest,  a  micro-electro- mechanical  oscillator.  All  major  theoretical 
predictions  have  been  fully  confirmed  in  the  experiment,  see  Fig.  1. 

6.  We  have  developed  a  theory  of  disease  extinction  in  large  populations  and  discovered  a 
new  scaling  behavior  of  the  extinction  rate  with  the  varying  reproduction  rate  of  infec¬ 
tion.  We  have  also  developed  a  mathematical  approach  that  allowed  us  to  predict  and 
describe  the  exponentially  strong  effect  of  random  vaccination  on  disease  extinction. 

7.  We  have  developed  a  theory  and  analysis  of  delayed  communication  in  stochastic 
swarms  to  examine  the  effect  of  latency  in  coordinated  behavior  of  multiple  vehicles. 
We  show  that  with  the  addition  of  a  time  delay,  the  model  possesses  a  transition  that 
depends  on  the  size  of  the  coupling  amplitude.  This  transition  is  independent  of  the 
initial  swarm  state  (traveling  or  rotating)  and  is  characterized  by  the  alignment  of  all 
of  the  individuals  along  with  a  swarm  oscillation.  See  Fig.  2. 
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Figure  1:  (a)  Switching  probability  distribution  in  a  parametrically  driven  micro-electro¬ 

mechanical  oscillator.  The  probability  distribution  P\2(X,  Y)  is  measured  for  switching  out  of 
state  A\  into  state  A2.  (b)  The  peak  locations  of  the  distribution  are  plotted  as  black  circles  and 
the  theoretical  most  probable  switching  path  is  indicated  by  the  red  line.  All  trajectories  originate 
from  within  the  green  circle  in  the  vicinity  of  A±  and  later  arrive  at  the  green  circle  around  A2. 
The  portion  of  the  distribution  outside  the  blue  lines  is  omitted. 

8.  We  show  that  steady-state  work  fluctuations  in  periodically  modulated  systems  dis¬ 
play  universal  features,  which  are  not  described  by  the  standard  fluctuation  theorems. 
Modulated  systems  often  have  coexisting  stable  periodic  states.  We  find  that  work 
fluctuations  sharply  increase  near  a  kinetic  phase  transition  where  the  state  popula¬ 
tions  are  close  to  each  other.  This  exponential  peak  is  a  new  strongly  pronounced 
phenomenon  which  has  not  been  previously  appreciated.  We  also  show  that  the  work 
variance  displays  scaling  with  the  distance  to  a  bifurcation  point  where  a  stable  state 
disappears  and  find  the  critical  exponent  for  a  saddle-node  bifurcation.  See  Fig.  3. 

9.  We  explore  the  distribution  of  paths  followed  in  fluctuation-induced  switching  between 
coexisting  stable  states.  We  introduce  a  quantitative  characteristic  of  the  path  distri¬ 
bution  in  phase  space  that  does  not  require  a  priori  knowledge  of  system  dynamics. 
The  theory  of  the  distribution  is  developed  and  its  direct  measurement  is  performed 
in  a  micromechanical  oscillator  driven  into  parametric  resonance.  The  experimental 
and  theoretical  results  on  the  shape  and  position  of  the  distribution  are  in  excellent 
agreement,  with  no  adjustable  parameters.  I11  addition,  the  experiment  provides  the 
first  demonstration  of  the  lack  of  time-reversal  symmetry  in  switching  of  systems  far 
from  thermal  equilibrium.  The  results  open  the  possibility  of  efficient  control  of  the 
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Figure  2:  Snapshots  of  a  swarm  taken  at  (a)  t  =50,  (b)  t  =60,  (c)  t  =62,  (d)  t  =64,  (e)  t  =66,  (f) 
t  =68,  (g)  t  =70,  (h)  t  =72,  (i)  t  =74,  and  (j)  t  =76,  with  N=300  particles,  and  noise  intensity, 
D=0.08.  The  swarm  was  in  a  rotational  state  when  the  time  delay  of  1  was  switched  on  at  t  =40. 


Figure  3:  Scaled  ratios  of  the  partial  work  variance  to  mean  partial  work  for  fluctuations  about 
periodic  attractors  of  a  resonantly  modulated  Duffing  oscillator,  as  functions  of  the  reduced  squared 
modulation  amplitude  /3.  The  curves  1  and  2  refer  to  large-  and  small-amplitude  vibrations,  re¬ 
spectively.  The  functions  S\  2  diverge  at  the  corresponding  bifurcation  points  of  the  oscillator. 


9 


1.5 


Figure  4:  Phase  portrait  of  a  two-variable  system  with  two  stable  states  A\  and  A 2.  The  saddle 
point  S  lies  on  the  separatrix  that  separates  the  corresponding  basins  of  attraction.  The  thin  solid 
lines  show  the  downhill  deterministic  trajectories  from  the  saddle  to  the  attractors.  A  portion 
of  the  separatrix  near  the  saddle  point  is  shown  as  the  dashed  line.  The  thick  solid  line  shows 
the  most  probable  trajectory  that  the  system  follows  in  a  fluctuation  from  A\  to  the  saddle.  The 
most  probable  switching  path  (MPSP)  from  A±  to  A  2  is  comprised  by  this  uphill  trajectory  and 
the  downhill  trajectory  from  S  to  A2.  The  plot  refers  to  the  system  studied  experimentally,  an 
underdamped  nonlinear  parametrically  modulated  oscillator  with  the  modulation  frequency  close 
to  twice  the  eigenfrequency. 

switching  probability  based  on  the  measured  narrow  path  distribution.  See  Fig.  4. 

10.  Population  extinction  is  of  central  interest  for  population  dynamics.  It  may  occur  from 
a  large  rare  fluctuation.  We  find  that,  in  contrast  to  related  large-fluctuation  effects  like 
noise-induced  interstate  switching,  quite  generally  extinction  rates  in  multipopulation 
systems  display  fragility,  where  the  height  of  the  effective  barrier  to  be  overcome  in 
the  fluctuation  depends  on  the  system  parameters  nonanalytically.  We  show  that  one 
of  the  best-known  models  of  epidemiology,  the  susceptible-infectious-susceptible  (SIS) 
model,  is  fragile  to  total  population  fluctuations.  See  Figs.  5  and  6. 

11.  We  have  investigated  the  stochastic  extinction  processes  in  a  class  of  epidemic  models 
and  showed  that  the  effective  entropic  barrier  for  extinction  in  a  susceptibleinfected- 
susceptible  epidemic  model  displays  scaling  with  the  distance  to  the  bifurcation  point, 
with  an  unusual  critical  exponent.  We  make  a  direct  comparison  between  predictions 
and  numerical  simulations.  We  also  consider  the  effect  of  non-Gaussian  vaccine  sched¬ 
ules,  and  show  numerically  how  the  extinction  process  may  be  enhanced  when  the 
vaccine  schedules  are  Poisson  distributed.  See  Fig.  7. 
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Figure  5:  A  snapshot  of  the  probability  p(X)  near  the  extinction  plane  X2  =  0  for  the  SIS  model;  p 
is  quasi-continuous  in  X\/N,  with  X\  and  X2  being  the  total  numbers  of  susceptibles  and  infected, 
respectively,  and  N  being  the  charateristic  total  population.  The  data  of  simulations  refer  to 
pt  =  9,i?o  =  4,  p'  =  p/(p  +  k)  =  0.1,  where  p  is  the  birth-death  rate  and  Ro  is  the  infection 
reproduction  rate.  For  t  =  0  the  system  was  at  the  stabl;e  state  X^,  the  total  number  of  particles 
was  N  =  50.  (b)  Asymptotic  optimal  trajectories  for  extinction  for  p  — >  0  (solid  line)  and  p  =  0 
(dashed  line). 


Figure  6:  The  switching  exponent  Q  for  the  SIS  model  of  epidemics.  The  epidemics  extinction 
rate  is  IF  oc  exp(— Q).  The  solid  and  dashed  lines  show  the  results  for  the  birth-death  rate  p  — >  0 
and  p  =  0,  respectively.  The  data  points  are  obtained  from  the  numerical  solution  of  the  master 
equation  for  the  total  initial  populations  N  =  50  and  N  =  100,  which  made  it  possible  to  directly 
extract  the  exponent  Q. 
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Figure  7:  Extinction  factor  increase  as  a  function  of  vaccination  parameters  numerically 
computed  using  the  Monte  Carlo  simulation.  Parameters  shown  are  g  ^vaccination  ampli¬ 
tude  (percentage  of  susceptibles)  and  v  =vaccination  frequency  per  year.  The  population 
size  is  N  =  1000. 
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